This invention relates generally to the field of digital microscopes, and more specifically to slide scanning digital microscope systems and methods for the detection of blank fields while scanning all or a portion of a microscope slide specimen.
Since the development of the first working optical microscopes, magnified images have been used in many areas of scientific research. Lens improvements in the 18th and 19th centuries greatly improved the performance of conventional compound (i.e., multi-lens) microscopes in producing accurate images. The electron microscope, developed in the 20th century, allowed scientists to obtain images of structures far smaller than those observable with optical microscopes. However, the use of optical microscopes to observe specimens mounted on glass slides remains important in many fields, such as botany, microbiology, geology, and medicine.
The utility of conventional compound optical microscopes for examining slide-mounted specimens is compromised because the field of view (FOV)—the portion of the specimen visible to the user through the objective lens at the eyepiece at any given time—becomes smaller as magnification increases. There is an inverse relationship between the optical magnification used to view the specimen and how much of the overall slide may be seen in the FOV.
The limited FOV of conventional compound microscopes becomes problematic when a user views a specimen under low magnification to identify a target area (also referred to as a region of interest or ROI) for viewing at high magnification. Once the target area is identified, the user must switch objective lenses to a higher magnification and reacquire the target area at the higher magnification. However, because the FOV at the higher magnification is smaller, the target area—which may have been completely visible in the FOV at the lower magnification—may not be visible in its entirety at the higher magnification. In addition, because there is no cross-reference to identify the location of the FOV at the new (higher) magnification within the overall slide specimen or within the FOV at the lower magnification, the user may have difficulty locating or be unsure where the target area is located after switching objective lenses. Additionally, the user may miss important portions of the target, or may be unsure whether the image acquired at the higher magnification is actually part of the target area identified at the lower magnification. Essentially, the user's location within the “forest” of the overall slide becomes lost when focusing on the “trees” of a specific location at higher magnification.
If the user is a pathologist scanning for cancerous cells in a tissue sample of a patient, for example, this could result in a missed diagnosis and obvious risk to patient health. In different contexts, the user may miss other desired structures, such as a particular cell wall region in a botanical sample, a particular group of microbial cells, specific crystal structures in a geological sample, or an area of high white blood cell counts in a blood sample.
In the last twenty-five years, the use of automated microscopes to generate digital images for examining microscope slides has become increasingly common. A class of microscopes using a combination of optical and electronic image acquisition and processing techniques, known as whole slide imaging (WSI) microscopes, has seen extensive growth.
WSI microscopes are automated microscopes that use a camera with a digital image sensor (DIS) to capture magnified images that may be viewed on a computer monitor or screen. An objective lens optically coupled to the camera provides magnification of the specimen on the microscope slide. Objective lenses at various magnifications (e.g., 20×, 40×, 60×, or 80×) may be used to provide a desired magnification. A movable stage holding the microscope slide allows the specimen to be moved in a pattern of rows and/or columns, and images may be captured of all or part of the slide. WSI microscopes also include an illuminator having a light source to illuminate the slide specimen, and illuminator optics to direct the light through the specimen and into the objective lens. A computer, which may be coupled to the WSI microscope or included as part of a WSI microscope system, is used to control the acquisition of digital images.
Because WSI microscope images are intended to be displayed on a computer or television screen or monitor, WSI microscopes typically lack the ocular or eyepiece lens in a standard compound microscope. Thus, magnification of the slide specimen is usually provided by a single objective lens (which may include a relay lens for infinity-focused objectives) in optical communication with the DIS. Stated differently, the optical path of a WSI microscope typically replaces the ocular lens with a digital image sensor (DIS).
WSI microscopes allow a user to deal with the FOV problem of conventional microscopes by providing computer-enabled features not present in standard compound ocular microscopes. These include a digital cross-reference of the “instantaneous” FOV of the objective lens at any time. Using this feature, the user can always see (e.g., by digital cross-hairs or other highlighting) where the instantaneous objective lens FOV is located within the overall slide specimen. More importantly, WSI microscopes allow the user to view a relatively large target area or region of interest (ROI) at high magnification in its entirety. And like the instantaneous view of the objective lens, the ROI is also cross-referenced to show the user at a glance its location within the overall slide specimen.
WSI microscopes typically provide an overview image of the entire specimen area of the slide at low (or zero) magnification in an overview image window on a computer screen. The overview image may be taken by an overview camera at low or zero optical magnification and displayed as a thumbnail image in the overview window. The digital cross-reference of the instantaneous objective lens FOV to the overview image is obtained by calibration during the manufacturing process. Typically, a calibration slide having a grid and/or other features (e.g., circles) of known dimensions provides a coordinate system that may be displayed on the computer screen. The objective lens is then centered on several key positions on the calibration slide, and the computer is used to store the locations (i.e., the X, Y coordinates of the side stage) of those positions within a memory (which may be in the WSI microscope or an attached computer). By storing the location of a number of different positions on the calibration slide, the WSI can always determine the location of where the objective lens is currently positioned on the slide. Equally importantly, the computer can reposition the objective lens to any given point designated by the user on the overview image, i.e., the user can cause the objective lens to “jump” to a new location on the slide by designating its position (e.g., with a mouse or touchscreen) on the overview image. The jump (new instantaneous) position may then be identified in the overview image by digital crosshairs, highlighting, etc.
In some microscopes, the overview image may be digitally enlarged by adding additional pixels to create a larger image. The additional pixels, however, add no additional image detail. Thus, the image appears large on the screen, but the image resolution remains the same. The digitally enlarged (low quality) overview image may assist the user in designating the one or more target or ROI areas for higher magnification and resolution.
While useful, however, knowing the “instantaneous” position of the objective lens FOV within the overview image is less important than being able view a relatively large target area or region of interest (ROI) in its entirety (i.e., as a single image which may be scrolled through on a computer screen or monitor). WSI microscopes allow a user to obtain a single image of a target area that is much larger (tens, hundreds, or thousands of times the area of a single objective lens FOV) for review within a window of a computer screen. This mode of operation, sometimes referred to as “scan” or “zoom” mode), allows the user to work with an image that is much smaller than the entire slide, and helps to ensure specific areas for review at higher magnification are not missed because of the FOV problem. In addition, the target area/ROI can also be cross-referenced and indicated (e.g., as a highlighted box or other area) on the overview image.
To generate an ROI image at high magnification, the user first designates on the overview image, using, e.g., a mouse or touchscreen, a target area of the specimen. After the ROI is designated and highlighted, the user may then cause the WSI microscope to scan the ROI and create a series of smaller, overlapping digital images of adjacent Fields of View (each of which is usually referred to as a “field image” or “tile”) that completely cover the target area (ROI). The overlapping field images are acquired by the digital camera as the movable stage moves the target area across the objective lens. Software then uses the overlapping edge areas of adjacent field images scanned by the digital camera to digitally combine the individual FOV images into a single high-magnification image of the ROI.
The single, high magnification ROI image may be viewed in its entirety within a window of a computer screen (e.g., by scrolling or standard computer window manipulations). The calibration process described above also allows the high magnification ROI image to be highlighted on the overview image (e.g., as a colored or darkened box, etc.). A variety of software algorithms may be used to combine the field images. The resulting composite, magnified ROI image is similar to a panoramic image assembled by software from a series of overlapping images on non-microscopic digital cameras.
Software may also be used to create an artificial “zoom in/zoom out” ability when viewing the complete ROI image. A series of images at various lesser magnifications of the single, fully magnified ROI image are created by image processing algorithms to digitally reduce the detail level in the fully magnified ROI. Although the level of detail may be reduced, it enables the user to “zoom out” to see a larger portion of the entire slide, up to and including the entire specimen area. These intermediate magnifications may be accessed by the user, e.g., using a mouse wheel or keystrokes, when viewing the complete ROI image comprising multiple field images.
In one mode of operation (“browse mode” or “live view mode”), the user may view or browse any area of the specimen through the instantaneous objective lens FOV (e.g., at 20×, 40×, 60×, etc.). In browse mode, a window is used to display the “live” field of view currently received by the digital image sensor from the objective lens. The overview image may be displayed in a corner of the browse window or in a smaller thumbnail window. The overview image includes highlighting (e.g., cross-hairs, a highlighted dot or box) to visually designate where the current FOV image of the objective lens (i.e., the image in the browse window) is located within the overview image.
The browse/live view mode thus maps the current FOV of the objective lens back to the overview image of the specimen. In doing so, the user is provided with a visual indication of which “trees” (browse window image) within the slide “forest” (the overview) are being viewed at high magnification in browse mode at any given time. In WSI microscopes at higher magnifications (e.g., 40× or 60×), the FOV of the objective lens may be so small that visualizing it within the overview image is enhanced by digitally enlarging the overview image in the overview image window, so that the objective lens FOV can be seen as a box or area on the overview image rather than a single point.
In some WSI microscopes, separate windows may be used for each of the overview, browse, and zoom/ROI images, while in other systems the overview image may be includes as part of the larger screen image in either browse mode operation or zoom mode operation. In general, the user may toggle between the overview and/or browse or zoom images (i.e., between a low-magnification and a high-magnification image. Typically, the relative sizes of the overview (low-magnification) and high-magnification images (i.e., the browse/“live FOV” image or a zoom/target area/ROI image).
WSI microscopes having the foregoing functions and features are described in, e.g., U.S. Pat. Nos. 6,101,265 and 6,711,283, each of which is hereby incorporated by reference in its entirety. Compact WSI microscopes having similar features to those described are available from Microscopes International, LLC (Plano, Tex.) with a 20×, 40× or 60× magnification. Models include the uScope MXII, the uScope HXII, the uScope DXII, and the uScope GXII.
Although there are significant variations among commercially available systems, a WSI microscope typically includes a movable stage that holds the microscope slide. In some systems, the stage is motorized to move at a constant speed, and digital field images are taken while the stage is moving, at time intervals synchronized to the stage speed to obtain field images for combination into the high-resolution ROI image. These WSI systems are referred to herein as moving image acquisition (MIA) systems. Conversely, in other systems the stage is motorized to move rapidly to a series of fixed or stationary positions from which the field images of the ROI are captured and subsequently combined. These WSI systems are referred to herein as fixed image acquisition (FIA) systems, because the image is obtained with the stage in a fixed (i.e., stationary) position. In both MIA and FIA systems, one or more motors are typically provided to move the stage in and out (X axis) and left and right (Y axis).
WSI microscopes also include an illumination system providing light to the slide stage, and an objective lens to magnify the light from the slide specimen and focus it on a digital image sensor (DIS) element in a digital camera. Focusing is typically provided by making one or more of the stage and the camera/objective lens structure (the camera and objective lens in WSI microscopes are typically coupled to a tube to maintain a fixed distance therebetween) movable by a motor (Z axis) capable of finely controlled, small movements on an axis generally perpendicular to the slide stage. This allows structures at different depths within the specimen to be captured in proper focus.
Digital image sensor cameras for WSI microscopes typically involve a CCD (charge coupled device) or a CMOS (complementary metal oxide semiconductor) image sensor as the DIS element. In some WSI systems, the DIS is incorporated into a static (snapshot) camera that captures field/tile images and combines them into a single, high-definition ROI image. In other systems, the DIS is incorporated into a video camera that outputs a stream of video images, from which frames may be captured as still image fields/tiles. Recent trends in digital photography, particularly digital video cameras, have seen a migration toward CMOS digital image sensors, which are most cost-effective than CCD sensors.
For video cameras, each distinct image output by the DIS element is known as a frame. Output rates for such video cameras are measured in frames per second (FPS), with typical rates between 10 and 90 FPS. Thus, at a frame rate of 30 FPS (typical for many video cameras), each new image in the video stream is created every 33.33 milliseconds (mSec), which is referred to as the frame time (FT). Similarly, at 50 FPS, the images are captured every 20 mSec. For FIA systems, the images are obtained while the stage is not moving, so any video frame captured while the stage is stationary may be used as the field image or tile.
Once all the field images comprising a ROI have been captured, image combination algorithms are used to combine the field images into a single, high-definition ROI image. Although a variety of image combination algorithms are used, one class of algorithms known as pattern matching algorithms operates by mathematically aligning the edges of adjacent field images until they overlap, and then combining the images at the overlapping region.
A significant limitation associated with WSI microscopes is the time required to take and compile the field images or tiles for combination into the single high definition ROI image. To obtain a ROI or target area image using either a MIA or FIA system, the WSI microscope objective is moved over the ROI, and magnified images are taken of very small areas either while the stage is moving (MIA systems) or stationary (FIA systems). The X-axis and Y-axis (stage movement) and Z-axis (focus) motors in the WSI microscope must be synchronized with the camera to properly capture digital field images.
In FIA systems, each X-axis and Y-axis movement to a new stationary field position takes a certain time (move time or MT) to occur, and causes a system vibration that requires a certain damping time (Settle Time or ST) to elapse before a usable field image may be taken (i.e., a field image without blurring). Settle times are imposed as a delay period after the slide stage is moved to a new field position. The ST delay period enables the vibrations associated with the movement (and the stopping of the movement) to dampen out, and thereby ensures that a field image captured by the SSM camera is a stable image without blurring. Imposing a post-movement Settle Time ensures that the field image is suitable for use as a field or tile in a combined ROI or target area image. Z-axis movements of the objective lens to obtain images at different focuses also require a move time, and a Z-axis (focus movement) ST may also be required before a usable image may be captured. Thus, Frame Times are limited not only by the move time in moving the camera from a first image position to a second image position, but also by the settle time necessary to resolve the vibrations following the move and enable a usable image to be captured.
Furthermore, after the settle time, any partially completed video frame output from the camera must be completed before the next video frame can be captured (frame completion time or FCT).
For some field images, called exhaustive focus fields (EFF), multiple images at the same slide stage position (i.e., X-axis and Y-axis location) in FIA systems are taken at different focus points (Z-axis positions). This involves a Z-axis focusing movement of either the slide stage or the light path/tube containing the DIS and the objective lens. In some cases, an additional settle time (ST) may be necessary for the camera vibration associated with the Z-axis movement to dampen out. In such cases, and additional FCT period may be added to complete the video frame output occurring when the Z-axis settle time elapses.
In many instances, a specimen may have an irregular shape, and much of a target area may be blank (i.e., the field image may contain none of the microscope specimen). This is particularly significant when a relatively large target area/ROI having an irregular or filamentary structure, or which includes void areas within a larger image. If these areas could be rapidly identified, operation of the WSI microscope could be significantly speeded up. Blank fields are particularly time-consuming for exhaustive focus fields, because the WSI microscope captures numerous wasted field images as part of a Z-stack (i.e., a group of differently-focused images at a single location) in a futile attempt to obtain an image with an improved focus score. However, since there is usually little data in the field image other than variations associated with dust, dirt, debris, and slight variations in lighting across the field image, the control unit may cause the objective lens to move through a wide range of focus positions (Z-axis movements) in an attempt to obtain a “best image” for a field that actually has no specimen content.
The process of capturing the multiple field images in a Fixed Image Acquisition (FIA) system for compilation into the ROI image includes the following steps:
1. move the slide stage to a desired (X, Y) location relative to the objective lens (slide stage MT);
2. wait for the vibration from the stage motion to dampen out (slide stage ST);
3. wait for the current partially completed video frame to finish (FCT);
4. capture the next complete video frame output from the camera (FT);
5. calculate a focus score for the captured field image;
6. change the focus position (Z axis location) of the objective lens at the same X, Y location (optional step for multiple images at the same field position to obtain the best focus; Z-axis MT);
7. wait for the focus (Z-axis) movement to dampen out (optional step for multiple images at the same field position to obtain the best focus; Z-axis ST);
8. wait for the current partially completed frame to finish (optional step for multiple images at the same field position to obtain the best focus; FCT);
9. capture the next complete image output from the camera (optional step for multiple images at the same field position to obtain the best focus; FT);
10. calculate a focus score for the captured field image
11. repeat steps 6-10 for multiple images until all images at the same field position are obtained; the multiple images at the same location are referred to as a Z stack, and the image from the Z-stack having the most appropriate focus score is retained as the (single) image for the field position;
12. repeat steps 1-5 (for single-focus images) or 1-11 (for exhaustive focus fields) until field images for all desired the ROI have been scanned.
Additional detail about existing WSI systems is provided in co-pending U.S. patent application Ser. No. 15/616,922, filed Jun. 8, 2017, entitled “Systems and Methods for Rapid Scanning of Images in Digital Microscopes,” which is hereby incorporated by reference in its entirety.
Ideally, all movements (X-axis, Y-axis, and Z-axis) would occur instantaneously, and with no settle time. This would allow each field image captured from the camera's video stream to have a valid, usable image. At 50 FPS, the camera would produce 50 field images each second. In addition, all blank fields would preferably be identified either in advance of the scanning process and not scanned at all, or identified rapidly during scanning so that the slide stage may rapidly be moved to another position.
In Fixed Image Acquisition (FIA) systems, the settle time for the stage (or objective lens) movement substantially slows the field image acquisition process for preparing a ROI image. Moreover, for irregularly shaped specimens with a relatively high number of blank fields, the process of scanning the blank fields imposes an additional time burden for acquiring field images. There is a need for improved systems with faster field image acquisition for combining into an ROI image.